In order to assist in the fast and efficient diagnosis of breast cancer and provide the prognosis information for pathologists, a computer-aided diagnosis approach for automatically grading breast pathological images was proposed. In the proposed algorithm,cells of pathological images were first automatically detected by deep convolutional neural network and sliding window. Then, the algorithms of color separation based on sparse non-negative matrix factorization, marker controlled watershed, and ellipse fitting were integrated to get the boundary of each cell. A total of 203-dimensional image-derived features, including architectural features of tumor, texture and shape features of epithelial cells were extracted from the pathological images based on the detected cells and the fitted boundary. A Support Vector Machine (SVM) classifier was trained by using the extracted features to realize the automated grading of pathological images. In order to verify the proposed algorithm, a total of 49 Hematoxylin & Eosin (H&E)-stained breast pathological images obtained from 17 patients were considered. The experimental results show that,for 100 ten-fold cross-validation trials, the features with the cell shape and the spatial structure of organization of pathological image set successfully distinguish test samples of low, intermediate and high grades with classification accuracy of 90.20%. Moreover, the proposed algorithm is able to distinguish high grade, intermediate grade, and low grade patients with accuracy of 92.87%, 82.88% and 93.61%, respectively. Compared with the methods only using texture feature or architectural feature, the proposed algorithm has a higher accuracy. The proposed algorithm can accurately distinguish the grade of tumor for pathological images and the difference of accuracy between grades is small.